fix(ai): resolve catalog-dependent test models at runtime instead of pinning ids - #7
Merged
Merged
Conversation
…pinning ids CI regenerates models.generated.ts from the live models.dev catalog before tsgo and vitest run, so tests pinning catalog ids break whenever the catalog moves with no repo change. The current revision dropped workers-ai/@cf/moonshotai/kimi-k2.6 from the cloudflare-ai-gateway listing and accounts/fireworks/routers/kimi-k2p6-turbo from fireworks (eight TS2345 sites plus a runtime TypeError where getModel returned undefined into streamSimple), and repriced moonshotai/kimi-k3 (3 -> 3.45), failing an exact cost assertion. Same approach the earlier claude-sonnet-4.5 rename fix established: resolve the model from the generated catalog at runtime and skip when absent. - kimi-test-model.ts: add getCloudflareGatewayWorkersAiTestModel(), which picks the newest workers-ai /compat model from the cloudflare-ai-gateway catalog, preferring Kimi ids - stream/empty/tokens/tool-call-without-result/total-tokens/ unicode-surrogate: the gateway suites use the resolver and add it to their credential skipIf - openai-completions-empty-tools: the two mock-backed /compat tests use the resolver behind it.skipIf, removing the undefined-model TypeError - fireworks-models: the router test resolves the current accounts/fireworks/routers/ entry and asserts the generator invariants (api, baseUrl, text input) instead of a pinned id and live modalities - prime-inference-models: assert kimi-k3 cost shape (positive input/output) instead of exact prices models.dev controls Validated: npm run check passes; the three runnable test files pass 20/20; the six live-suite files collect cleanly and skip without credentials. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BiT6ArSmvtdUbKEZgGAzrA
The resolver narrowed against cloudflare-ai-gateway's current api union via a type predicate and an api comparison. That union is itself regenerated from the live catalog, so when CI's regeneration dropped every workers-ai /compat entry the predicate target, the comparison, and the return type all became provably impossible (TS2677/TS2367/TS2322) — the same catalog-shape dependence this branch removes, one level up. Type the intermediate list as Model<Api>[] (pure widening, valid under any catalog), filter by the workers-ai/ id prefix, and narrow only the final result with a cast that the generator's construction guarantees: every workers-ai/ gateway entry is an openai-completions /compat route. Validated in both catalog shapes: with the checked-in catalog, npm run check passes and the three runnable test files pass 20/20; with the workers-ai gateway entries stripped to mirror CI's regenerated catalog, tsgo passes and all gateway suites plus the two mock-backed /compat tests skip cleanly (18 passed, 460 skipped, 0 failed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BiT6ArSmvtdUbKEZgGAzrA
thomaswillner
marked this pull request as ready for review
August 28, 2026 02:46
thomaswillner
added a commit
that referenced
this pull request
Aug 28, 2026
…n input (#6) Adds three documents under orchestration/spx-v2/: the claim-by-claim challenge of the external SPX V2 audit with file:line evidence and the V1 coverage verdict; the corrected, agent-portable implementation input of record (binding IBKR PAPER/LIVE requirements R1-R3, CBOE EOD-only supersession, ordered slices S0-S8 mapped to open issues, method contract, guardrails, open decisions D1-D8); and the session self-refinement record. Also carries the ported fix(ai) catalog-drift commits, which no-op against the base now that #7 is merged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BiT6ArSmvtdUbKEZgGAzrA
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f012e79668
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const gatewayModels: Model<Api>[] = getModels("cloudflare-ai-gateway"); | ||
| const models = gatewayModels.filter((model) => model.id.startsWith("workers-ai/")); | ||
| const kimis = models.filter((model) => model.id.includes("/moonshotai/kimi-")); | ||
| const pool = kimis.length > 0 ? kimis : models; |
There was a problem hiding this comment.
When a regenerated catalog contains Workers AI routes but no Kimi route, this fallback may select a model with reasoning: false. The Cloudflare Gateway block in stream.test.ts then unconditionally runs handleThinking, which requires thinking events and will fail despite the selected model not supporting them. Filter the fallback on model.reasoning, or skip the thinking-specific test when the resolved model lacks reasoning support.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI regenerates
packages/ai/src/models.generated.tsfrom the live models.dev catalog beforetsgoand vitest run, so tests that pin catalog ids go red whenever the catalog moves — with zero repo changes. That is what brokeBuild and checkandTest (ai)on #6 (a docs-only PR) and would fail identically onmaintoday:workers-ai/@cf/moonshotai/kimi-k2.6vanished from thecloudflare-ai-gatewaylisting → 8 TS2345 sites acrossempty,openai-completions-empty-tools,stream,tokens,tool-call-without-result,total-tokens,unicode-surrogate, plus a runtimeTypeError: Cannot read properties of undefined (reading 'api')wheregetModel(...)!fedstreamSimple.accounts/fireworks/routers/kimi-k2p6-turbovanished from fireworks (catalog now listsrouters/kimi-k3-fast) → TS2345 infireworks-models.moonshotai/kimi-k3repriced (3 → 3.45) → exact cost assertion failed inprime-inference-models.This is the same recurring mode
760ccd8fixed for theclaude-sonnet-4-5→claude-sonnet-4.5rename.Fix (same pattern as
760ccd8: resolve from the catalog, skip when absent)kimi-test-model.ts: newgetCloudflareGatewayWorkersAiTestModel()— picks the newestworkers-ai//compat model from thecloudflare-ai-gatewaycatalog, preferring Kimi ids so the exercised model stays comparable; documented as skipIf-guarded.stream,empty,tokens,tool-call-without-result,total-tokens,unicode-surrogate): the gateway describe blocks use the resolver and add|| !modelto their credentialskipIf.openai-completions-empty-tools: the two mock-backed /compat tests (which always run in CI) use the resolver behindit.skipIf, eliminating the undefined-model TypeError.fireworks-models: the router test resolves the currentaccounts/fireworks/routers/entry and asserts generator invariants (api: anthropic-messagesand the baseUrl are hardcoded for every fireworks entry ingenerate-models.ts;inputmirrors live modalities, so only its text floor is asserted).prime-inference-models: kimi-k3 cost assertions relaxed to shape (positive input/output) — exact prices are models.dev's to change.No production code touched;
models.generated.tsuntouched.Validation
npm run checkpasses (biome, tsgo, installer render, browser smoke) — also re-ran green in the husky pre-commit hook.openai-completions-empty-tools,fireworks-models,prime-inference-models.AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY— absent in CI, unrelated to this diff.Once this merges, #6 (and any future PR) stops inheriting these two red checks; the same commit is also ported onto #6's branch so it can go green independently, and it no-ops when the base carries this change.
🤖 Generated with Claude Code
https://claude.ai/code/session_01BiT6ArSmvtdUbKEZgGAzrA
Generated by Claude Code